Skip to content

ENH: Add method='table' for EWM.mean #42273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 29, 2021
Merged

ENH: Add method='table' for EWM.mean #42273

merged 4 commits into from
Jun 29, 2021

Conversation

mroeschke
Copy link
Member

  • tests added / passed
  • Ensure all linting tests pass, see here for how to run them
  • whatsnew entry

@mroeschke mroeschke added this to the 1.3 milestone Jun 28, 2021
@mroeschke mroeschke added Enhancement Window rolling, ewma, expanding labels Jun 28, 2021
jreback
jreback requested changes Jun 28, 2021
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any asv's hit by this? or can you just show the perf update

@jreback jreback merged commit bba53fc into pandas-dev:master Jun 29, 2021
@jreback
Copy link
Contributor

jreback commented Jun 29, 2021

@meeseeksdev backport 1.3.x

@jreback
Copy link
Contributor

jreback commented Jun 29, 2021

thanks @mroeschke

@meeseeksmachine meeseeksmachine mentioned this pull request Jun 29, 2021
Closed
meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Jun 29, 2021
@mroeschke @meeseeksmachine
Backport PR pandas-dev#42273: ENH: Add method='table' for EWM.mean
d1595d7
@lumberbot-app
Copy link

lumberbot-app bot commented Jun 29, 2021

Something went wrong ... Please have a look at my logs.

simonjayhawkins added a commit that referenced this pull request Jun 29, 2021
@simonjayhawkins
Revert "ENH: Add method='table' for EWM.mean (#42273)"
3ead2e1 
This reverts commit bba53fc.
@simonjayhawkins simonjayhawkins mentioned this pull request Jun 29, 2021
Merged
@simonjayhawkins
Copy link
Member

This is failing ci. reverting to get ci back to green

@mroeschke can you open a new PR for this.

simonjayhawkins added a commit that referenced this pull request Jun 29, 2021
@simonjayhawkins
Revert "ENH: Add method='table' for EWM.mean (#42273)" (#42302)
12513c4 
This reverts commit bba53fc.
mroeschke
mroeschke commented Jul 1, 2021
View reviewed changes
pandas/core/window/numba_.py
numba = import_optional_dependency("numba")

@numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)
def ewma_table(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stuartarchibald if you have time, could you review if this function looks if I have any bad practices in this function. Some context:

  1. Generally this is run with @numba.jit(nopython=True, nogil=False, parallel=False)
  2. This function has variables from a global scope com, adjust, ignore_na, deltas
  3. I cache this function in a dictionary this first time it runs (in NUMBA_FUNC_CACHE) and later calls will return the cached fuction.

When using this function, sometimes I get a segfault:

(pandas-dev) matthewroeschke@x86_64-apple-darwin13 pandas-mroeschke % ipython
Python 3.8.6 | packaged by conda-forge | (default, Dec 26 2020, 04:50:20)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.25.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: df = DataFrame(np.random.rand(3, 30))

In [2]: ewm = df.ewm(com=1, method="table")

In [3]: ewm.mean(engine="numba")
Out[3]:
         0         1         2         3         4   ...        25        26        27        28        29
0  0.770949  0.396205  0.728331  0.417464  0.715142  ...  0.237903  0.383182  0.021471  0.837933  0.590543
1  0.417231  0.369916  0.793127  0.961175  0.531131  ...  0.145222  0.152615  0.714216  0.515145  0.238283
2  0.441215  0.220925  0.868065  0.736689  0.858115  ...  0.165413  0.977209  0.549496  0.472534  0.555430

[3 rows x 30 columns]

In [4]: ewm.mean(engine="numba")
Out[4]:
         0         1         2         3         4   ...        25        26        27        28        29
0  0.770949  0.396205  0.728331  0.417464  0.715142  ...  0.237903  0.383182  0.021471  0.837933  0.590543
1  0.417231  0.369916  0.793127  0.417464  0.531131  ...  0.145222  0.152615  0.714216  0.515145  0.238283
2  0.441215  0.220925  0.868065  0.417464  0.858115  ...  0.165413  0.977209  0.549496  0.472534  0.555430

[3 rows x 30 columns]

In [5]: ewm.mean(engine="numba")
Out[5]:
         0         1         2         3         4   ...        25        26        27        28        29
0  0.770949  0.396205  0.728331  0.417464  0.715142  ...  0.237903  0.383182  0.021471  0.837933  0.590543
1  0.417231  0.369916  0.793127  0.961175  0.531131  ...  0.145222  0.152615  0.714216  0.515145  0.238283
2  0.441215  0.220925  0.868065  0.736689  0.858115  ...  0.165413  0.977209  0.549496  0.472534  0.555430

[3 rows x 30 columns]

In [6]: ewm.mean(engine="numba")
python(3345,0x111f53dc0) malloc: Incorrect checksum for freed object 0x7fd3255c55a8: probably modified after being freed.
Corrupt value: 0x3ff0000000000000
python(3345,0x111f53dc0) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort      ipython

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke I'll take a look. Did you already try running it a) without the @jit decorator and b) with @jit(boundscheck=True) ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the noise @stuartarchibald. I found a bug in my algorithm. I should have tested without the @jit decorator first.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke no worries. Numba also has this environment variable to switch off JIT compilation without needing to modify code: https://numba.readthedocs.io/en/stable/reference/envvars.html#envvar-NUMBA_DISABLE_JIT, it might be useful for helping with development/testing.

JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@mroeschke @JulianWgs
ENH: Add method='table' for EWM.mean (pandas-dev#42273)
f54eaa7
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@simonjayhawkins @JulianWgs
Revert "ENH: Add method='table' for EWM.mean (pandas-dev#42273)" (p…
c3e96f7 
…andas-dev#42302)

This reverts commit bba53fc.
@jreback jreback mentioned this pull request Jul 6, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stuartarchibald stuartarchibald stuartarchibald left review comments

@jreback jreback jreback requested changes

Assignees
No one assigned
Labels
Enhancement Window rolling, ewma, expanding
Projects
None yet
Milestone
1.3
Development

Successfully merging this pull request may close these issues.
4 participants
@mroeschke @jreback @simonjayhawkins @stuartarchibald